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Uses of Student Outcome Indicators at the 
Provincial and National Levels: 
Issues and Solutions 


The last decade has seen an enormous increase in interest in and use 
of educational indicators. Student learning is the fundamental 
purpose of education. This paper focuses on K-12 student outcome 
indicators and measures of how well students have learned what is 
expected of them — the raison d'étre of schooling. It provides a 
Canadian perspective on student outcome indicators and has three 
major purposes: (1) to identify types of student outcome indicators 
and sources of information, (2) to discuss provincial and national 
uses, and (3) to discuss issues and identify solutions. A March 1995 
survey of provinces and territories provides the information base for 
current Canadian practice in using and addressing issues in student 
outcome indicators for this paper. 


This paper focuses on student outcome indicators because they address the 
fundamental purpose of schooling, namely, that students leam. These indicators 

are expected to be part of an integrated system which describes a broad range of 

the conditions of schooling (context, inputs, processes) and desired outcomes. An 
indicator system should be rooted in a conception of the important components of 
the operation and outcomes of schooling, and be based on the goals, priorities, and 
expectations of its originators. It should consist of multiple goals of education 
(based on appropriate dimensions and domains of schooling), multiple indicators 

of each goal measured by multiple methods, multiple levels of analysis, and multiple 
participants (McEwen & Chow, 1991). 


There is an extensive body of literature on indicator systems as a tool for reform and 
accountability. For example, an annotated bibliography prepared for Alberta's 
Educational Quality Indicators initiative (Alberta Education, 1990) contained 350 
Citations organized into 14 themes under three major headings: indicator systems, 
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interpretative framework, and outcomes. Much has been written on the subject 
since. Making Education Count (Organization for Economic Cooperation and 
Development [OECD], 1994), for instance, is a useful resource on the organization, 
development, measurement, and uses of indicators, and Accountability in Education 
in Canada! (McEwen, 1995) provides a Canadian perspective. 


Student Outcome Indicators 
There are different types or student outcome indicators and sources of information 
for provincial and national uses. 


Tvpes 

There are essentially three types of student outcome indicators - cognitive, 
affective, and behavioral — based on desired intellectual, personal, social, and 
vocational learning found in most provincial statements of goals of schooling. 
Despite the fact that most educational goal statements include variations of 

this broad range of desired outcomes, student outcome indicators tend to focus 
on the cognitive domain, specifically academic achievement. Indeed indicator 
systems are often criticized for including too narrow a range of outcome 
indicators. Affective indicators, less prevalent than cognitive indicators, are 
usually some measure of attitude. Fewer provinces use this type of information 
because of the difficulties in measuring affective outcomes. Some people also 
consider affective measures to be beyond the purview of the school. Behavioral 
outcomes are often negatively stated - dropout rates, absenteeism, etc. 


Table 1 presents the results of the March 1995 survey to identify the most common 
student outcome indicators used by Canadian provinces and territories. (Province 
will be used hereafter to refer to both provinces and territories.) Achievement and 
participation are the two most commonly used indicators; satisfaction and attitudes 
are the least used. Prince Edward Island is the only province that does not currently 
use student outcome indicators. 
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Table 1 
Types of Student Outcome Indicators Used in Canada 


Province/Territory Achievement Participation Graduation Dropout Satisfaction Attitudes 


British Columbia v v v v v 
Alberta v V v v v 
Saskatchewan Vv v v v V 1 
Manitoba Vv v v v 

Ontario v V v v v v 
Quebec Vv V v v 

New Brunswick v v v v v v 
Nova Scotia v v v v 

Newfoundland Vv v v v v q 
Northwest Territories V v v v 

Yukon v v 

Total 11 Il 10 8 7 6 
Sources of Information 


The three most common ways to collect student data are assessment programs, 
administrative data, and surveys. A survey of student assessment and examination 
programs in Canada (Alberta Working Group, 1994) revealed that all but two 
jurisdictions (Prince Edward Island and the Northwest Territories) had some type of 
provincial assessment program in the 1993/94 school year. Nine jurisdictions also 
reported having high school exit examinations for graduating students; the stakes 
for students range from 30 to 50 percent of the final mark in exam courses, with the 
remainder assigned by teachers for class work. Traub (1994) and the United States 
General Accounting Office (1993) also provide useful information about provincial 
testing programs. All educational jurisdictions collect administrative data about 
their students - enrollment, gender, types of program, and so forth. These data can 
be manipulated to arrive at selected indicators such as participation in specific 


Student Outcome Indicators 3 6 Uses and Issues 


programs or in school, and graduation and dropout rates. Finally some jurisdictions 
collect perceptual data from students and others through surveys; such qualitative 
information helps in the interpretation of the quantitative information provided by 
assessments and administrative data, and provides some indication of affective 

and behavioral outcomes which are not readily available from assessments or 
administrative data. 


Each province produces an annual report on education. These reports contain 
information about the provincial department of education and typically include 
Statistics on students, teachers, and finances. Some provinces also have indicator 
systems which provide a broader range of information about their education 
systems. Quebec was the first province to publish an indicators report in 1986. 
The most recent addition is British Columbia which released its first report in 
1994, 


At the national level, there are three organizations which collect information about 
students: the Council of Ministers of Education, Canada (CMEC), Statistics 
Canada (StatsCan) and the Canadian Education Statistics Council (CESC). 


Council of Ministers of Education, Canada ~ CMEC provides the ministers 
responsible for education with a mechanism for consultation in matters of mutual 
interest and concem, and facilitates cooperation in a broad range of activities at 
the elementary, secondary, and post-secondary levels. In 1989 CMEC initiated 
the School Achievement Indicators Program (SAIP) which was the first attempt 
to arrive at a consensus on the elements of a national assessment. Ministers 
agreed to measure the achievement of 13-year-olds and 16-year-olds in reading, 
writing, mathematics, and science. The report on mathematics achievement was 
released in 1993 and the report on reading and writing in 1994 (CMEC, 1993, 
1994), 


Statistics Canada - Canada's federal statistical agency is responsible for collecting, 
compiling, analyzing, abstracting, and publishing statistical information relating to 
the commercial, industrial, financial, social, economic, demographic, and general 
activities and the condition of Canadians. Statistics Canada conducts a census every 
five years, develops and disseminates integrated social and economic statistics 
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pertaining to the whole of Canada and to each of the provinces, and coordinates 

plans for the integration of those statistics (Government of Canada, 1992). Statistics 
Canada has a number of publications which focus on education: Education Quarterly 
Review, the annual Education in Canada, A Statistical Review, and Educational 
Attainment and School Attendance, which contains information from the 1991 census. 


Canadian Education Statistics Council - CESC is a partnership between the 
CMEC and Statistics Canada. It produced its first Statistical Portrait of 
Elementary and Secondury Education in Canada in 1990; the second edition was 
released in 1992, and the third will appear in 1995. This publication contains 
information about students, educators, and finances (CESC, 1992). In September 
1993, CESC announced the initiation of the Pan-Canadian Education Indicators 
Program (PCEIP) to develop better measurements of the performance of our 
education systems. The long-term goal of this program is to create a full range of 
indicators to assess education and learning in Canada, from preschool to lifelong 
learning. The short-term goal is to develop statistical measures in selected 
indicator areas, focusing on all levels of education and training (elementary, 
secondary, and post-secondary). The six areas for priority development and the 
lead jurisdiction responsible for the work in consultation with all provinces and 
interested partners follow: (1) academic achievement — Alberta, (2) accessibility 
— Quebec, (3) student flows - British Columbia, (4) school/work transitions — 
Statistics Canada, (5) citizenship - Ontario, and (6) satisfaction - Newfoundland. 
The first conceptual paper on academic achievement was recently completed 
(Alberta Working Group, 1995). 


At the international level, three sources of information about education 

in Canada are the ongoing Indicators of Education Systems (INES) project, 
the International Assessment of Educational Progress (IAEP), and the Third 
Intemational Mathematics and Science Study (TIMSS) which will collect 
data this year. They are important because they provide both national and 
international comparisons of student achievement and other indicators. 


Indicators of Education Systems - Canada is participating in this international 


project through the CMEC. This OECD project includes four networks, each 
focusing on different aspects of education: (A) student outcomes, (B) student 


Student Outcome Indicators 5 8 Uses and Issues 


destinations, (C) school features and processes, and (D) attitudes and expectations. 
Canada is represented on Networks A and B. The INES report, Education at a 
Glance 2 (OECD, 1993), provided data on member countries in the following areas: 
demographic, economic, and social context; costs, resources, and school processes; 
and results of education. Statistics Canada provided provincial results for selected 
indicators, The third edition of this report was released April 11, 1995. 


International Assessment of Educational Progress - The first [AEP in 1988 assessed 
the mathematics and science achievement of 13-year-olds in five countries and four 
provinces (Lapointe, Mead, & Phillips, 1989). Nine provinces participated in LAEP II 
(Lapointe, Askew, & Mead, 1992; Lapointe, Mead, & Askew, 1992). In this 1991 
study, 20 countries assessed the mathematics and science achievement of 13-year-old 
students; 15 countries assessed a representative sample of students and five other 
countries participated in assessing special populations. Fourteen countries assessed 
samvles of their 9-year-old students in these subjects. About 175,000 9- and 13-year- 
olds were tested in 13 different languages. The study also provided the option of 
participating in a performance assessment. Four countries and five provinces 
participated in the performance task for 13-year-olds (Semple, 1992). The geography 
portion of IAEP II was a special study of the geographic knowledge and skills of 13- 
year-olds. Nine countries took part in this component. In Canada, eight provinces 
participated in the geography assessment (Lazer, 1992). 


Third International Mathematics and Science Study - The International Association 
for the Evaluation of Educational Achievement (IEA) is directing this study with 
more than 50 countries participating in the 1995 adrainistration. The data collection 
will focus on three areas — curriculum, instructional practices, and student outcomes. 
Students will be assessed at three levels: age 9 (grades 3 and 4), age 13 (grades 7 
and 8), and the last year in secondary school. The University of British Columbia 

is coordinating the Canadian national sample. Five provinces are participating as 
provincial samples: British Columbia, Alberta, Ontario, New Brunswick, and 
Newfoundland. 


Table 2 presents the major sources of information identified by respondents in 
the provincial survey. All provinces reported using Statistics Canada reports, 
followed by 11 for the SAIP reports (Saskatchewan is the only province not 
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participating in this program), ten reported using the CE:SC statistical portrait, and 
eight the second Intemational Assessment of Educational Progress (IAEP II) 
reports based on the assessments of mathematics, science, and geography 
conducted in 1991. 


Table 2 
Sources of Information for Indicators 


Province/Territory StatsCan! SAIP2 Portrait} AEP II¢ 


British Columbia V V V 
Alberta 
Saskatchewan 


Manitoba 


a a ae 


v 
v 
v 
Ontario V 
Quebec 


New Brunswick 


as 


Nova Scotia 
Prince Edward Island 
Newfoundland 


ee “wh. “Oz. Bs 


Northwest Territories 


Yukon 


we as, we vad as, ek eB eR, ow, ' “ 
a a a a 


Total 12 11 10 8 


1 Statistics Canada publications 

2School Achievement Indicators Program (mathematics, reading, writing) 
34 Statistical Portrait of Elementary and Secondary Education in Canada 
4international Assessment of Educational Progress II 
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Provincial and National Uses of Student Outcome Indicators 
Provinces and national associations use student outcome indicators to help them 
for various purposes. The media and others also use them. 


Provincial Uses 

Provinces use student outcome indicators for a variety of purposes. Table 3 
presents the most common purposes identified by the provinces. Eleven provinces 
reported using indicators to track treads over time. Ten indicated they use them 
for public accountability, and nine for informing policy and practice, decision 
making, and program evaluation. Eight provinces use them for program 
improvement and strategic planning, five use them to set a provincial standard, 
and three use them for school evaluation. . 


National Uses 

Education is a provincial responsibility in Canada. The Council of Ministers of 
Education, Canada facilitates interprovincial cooperation in the School Achievement 
Indicators Program which assesses achievement in the core subjects (to date, 
mathematics, reading, and writing). Provinces also participate in international studies 
of academic achievement; these studies provide both national and international 
comparisons. 


Provinces routinely use Statistics Canada's educational publications to Compare 
their performance on selected indicators with that in other provinces. Some also 
use the OECD reports for comparative purposes. National associations and the 
media use the interprovincial, national, and international data for describing the ~~ 
state of Canadian education and comparing it to that in other countries. 


Many organizations use secondary sources which are then widely quoted by the . 
media. Secondary sources are not always accurate. For example, Nagy (1994) 
discussed how an influential report, A Lot to Learn (Economic Council of Canada, 
1992) inappropriately interpreted international and national student achievement 
data which Freedman (1993) subsequently duplicated in a report aimed at parents. 
Among Nagy's criticisms are these authors’ misleading interpretation of Canadian’ 
achievement trends on the Canadian Tests of Basic Skills, and their distortion of 
international results by focusing on extreme groups. Barlow and Robertson's 
(1994) Class Warfare provides an international context for criticism of Canada’s 
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schools; these authors provide evidence to dispel a number of myths about the 
purported deterioration of public education in Canada. They specifically address 
the myths that our schools have failed us and our children?, that our graduates do not 
have the skills they need, and that big business is creating highly skilled jobs. These 
two publications illustrate how erroneous interpretations of data not only mislead the 
public, but can undermine confidence in education. 


The public gets most of its information about education from the media. A search of 
some of Canada's online newspaper and newswire data bases? indicated that in March 
1995, there were over 1,500 news items on education in Canada. If the international 
Reuter's Textline is added, more than 2,000 news items on education were available for 
print and electronic media to use in their coverage of education. In Alberta, newspapers 
published more than 1,200 articles on education. By contrast, Alberta Education 
released 72 press releases over the past twelve months, an average of about six per 
month. The Alberta example illustrates how the media present information about 
education at a rate far exceeding that provided by a provincial department of education. 


There is a clear need for the education community to provide accurate information 

that the lay reader can understand. Too often reports on education arc full of jargon 
which does not communicate. The public often prefers brief accounts of results and 
accepts the media's interpretation of public reports. Education officials should provide 
a synopsis to the media which will otherwise interpret the information according to their 
understanding of the material. Tufte (1983, 1990) has provided exemplars of how to 
display information for a visually literate public that often prefers to get its information 
from television rather than printed material. 


Issues and Solutions 

There are many issues in using student outcome indicators. This paper addresses 
political and financial, technical and administrative, interpretive and communicative, 
and comparative issues. There are several discussions of issues related to indicator 
systems in the literature; see, for example, Alberta Working Group (1995), McEwen 
and Chow (1991), Nagy (1994), Oakes (1986), Office of Educational Research and 
Improvement (1988), and Organization for Economic Cooperation and Development 
(1992, 1994). The issues and solutions are discussed together with examples from 
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the March survey to illustrate how provinces are addressing these issues. Four tables 
summarize the discussion of each set of issues; the 16 issues are numbered 
consecutively over the four tables. 


Political and Financial Issues 
There are four major political and financial issues: responsibility for education, 
value for investment, funding, and public confidence. 


Responsibility for Education - Section 93 of Canada's Constitution Act, 1867, 
confers on the provincial legislatures the exclusive power to make “laws in 
relation to education". By virtue of this power, the « stablishment and 
administration of schools and universities is a provincial responsibility (Hogg, 
1992, p. 1227). Therefore, each province is responsible for the governance, 
programs, and funding of its education system. At the national level, provinces 
cooperate through the Council of Ministers of Education, Canada and the 
Canadian Education Statistics Council on matters of mutual interest. 


Value for Investment - The value, or return on investment, of all public services 
is being questioned in this time of fiscal restraint, restructuring, and global 
competitiveness. In 1991/92, Canada invested $33 billion in elementary and 
secondary education, which accounts for 63% of the total national spending of 
$53 billion on all levels of education; an estimated 34% of the adult population 
held post-secondary credentials (Statistics Canada, 1994a). In the 1991 LAEP II, 
Canadian students scored above the average and ranked in the middle of 
participating countries in mathematics, science, and geography. Canada ranked 
first of 28 industrial countries on the 1992 human development index (HDI) 
which is a composite of longevity, knowledge, and standard of living* (United 
Nations Development Programme, 1994). However, the differences among the 
closely-clustered top countries are not significant (e.g., (1) Canada 0.932, (2) 
Switzerland 0.931, (3) Japan 0.929). See Finn (1994) and Castonguay (1995) 
for critiques of the HDI. 


Funding for Education - Notwithstanding the foregoing evidence of Canada’s return 


on its investment in education, provinces are facing reduced revenues, and, therefore, 
less money for education. At the K-12 level, they are prioritizing their activities and 


Student Outcome Indicators 11 Uses and Issues 


allocating resources to priorities. Nationally, they are cooperating through CMEC 
on curriculum development and on the School Achievement Indicators Program 
assessments in reading, writing, mathematics, and science for 13- and 16-year-old 
students. Through the CESC, they are involving national nongovernment 
organizations in the development of the Pan-Canadian Education Indicators 
Program. Regionally, some provinces are involved in cooperative curriculum 
development, namely, the Western Canadian Protocol for Collaboration in Basic 
Education (K-12) (language arts and mathematics) and the Atlantic Region Common 
Curriculum Initiatives (language arts, mathematics, and science, grades 1-12). 


Public Confidence — The provinces recognize the need to inspire confidence in 
public education since 57% of families do not have children who are less than 16 
years of age (Statistics Canada, 1994b). Studies have indicated that parents with 
children in school are more supportive of public education than people who are 

less directly involved in school (see, for example, Elem, Rose, & Gallup, 1994; 
Livingstone, Hart, & Davie, 1995). The proportion of families without children is 
increasing as Canadians age and have fewer children. It is imperative, therefore, for 
provinces to demonstrate the value of the investment in education so that the public 
has confidence that our next generation is well prepared to contribute to the social 
and economic well-being of Canada. Asking diverse clients - such as students, 
parents, and others — their opinions identifies strengths, weaknesses, and issues 
associated with public education. 


In the March survey, seven provinces indicated that they collect students’ perceptions; 
some also collect such qualitative information from other groups. The British 
Columbia Ministry of Education (1994) reported the attitudes of students, graduates, 
employers, and post-secondary instructors toward selected aspects of education; 
Alberta Education (1994b) has committed, in its three-year business plan, to 
measuring the satisfaction of students, parents, the public, educators, and the business 
community on a regular basis. Saskatchewan Education, Training and Employment 
(1994) reported the opinions of students, teachers, parents, and the public about the 
effectiveness of its education system. In 1994, Manitoba surveyed a sample of grade 
12 students to provide an evaluation of the province's education system and 
suggestions on how the system could be improved. Ontario collected perceptions 
from its grade 9 students, their teachers, and parents as part of its reading and writing 
assessment (Ontario Ministry of Education, 1994). The Ontario Institute for Studies 
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in Education has been conducting biennial surveys of public attitudes toward 
education in Ontario since 1978 (Livingstone, Hart, & Davie, 1995). New Brunswick 
surveys all of its grade 8 and 11 students on selected aspects of schooling, and 
Newfoundland surveys all of its students in grades 6, 8, and 12 about the quality of 
school life (Newfoundland Department of Education, 1994). Existing national 
surveys of attitudes toward K-12 education are either dated (Williams, & Millinoff, 
1990) or conducted for specific clients and, therefore, not readily accessible to the 
public. Satisfaction is one of the six areas for priority development in the Pan- 
Canadian Education Indicators Program. Table 4 summarizes the discussion of 
political and financial issues. 


Technical and Administrative Issues 

This involves the largest number of issues for student outcome indicators. It 
includes definitions, student comparability, census or sample, student data bases, 
and data collection and analysis. The absence of adequate infrastructure for 
tracking, monitoring, and compiling up-to-date data is a Canada-wide problem. 


Definitions - This is a major problem in national and international studies because 
definitions and methods used to calculate indicators are not uniform. Therefore, 
comparison and interpretation of indicators can be problematic. Dropout statistics 
serve as a useful illustration because they are so often cited as evidence of the 
failure of schools. Methods for calculating dropout statistics generally fall into 
three categories: annual, longitudinal, and net. 
¢ Annual rates estimate the proportion of all students who drop out of school 
in a given school year, 
¢ —_ longitudinal rates estimate the proportion of a given cohort of students 
who drop out over the course of their schooling (this includes former 
dropouts who returned to school), and 
* net rates estimate the proportion of a given cohort (e.g., 18-year-olds) 
who have not graduated and who are not in school at a specific point in 
time (this excludes former dropouts who have retumed to school). 


Gilbert et al. (1993) discussed five methods to estimate Canadian dropout rates 
which fall under the category of net rates: (1) the-;complement of the graduation 
rate, (2) the apparent cohort dropout rate, (3) census estimates, (4) Labour Force 
Survey estimates, and (5) the School Leavers Survey. The first two are based 
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Table 4 


Summary of Political and Financial Issues and Canadian Solutions 


1. Responsibility 
for education 


2. Value for 
investment 


3. Funding for 
education 


4. Public 
confidence 


Provincial Solutions 


Each province and territory 
is responsible for its own 
system of education. 


Provinces are becoming 
more proactive in 
demonstrating the value of 
public education through 
publications targeted at 
the public. 


In this time of fiscal 
restraint, provinces are 
prioritizing their activities 
and allocating scarce 
resources to priorities. 


Expectations of and 
satisfaction with education 
are important for public 
support. 


Several provinces are 


collecting perceptions 
about schooling from 


students and other groups 
es. a public, 


National Solutions 


Provinces cooperate 
through the Council of 
Ministers of Education, 
Canada and the Canadian 
Education Statistics 
Council on educational 
matters of mutual interest 
and concern. 


Through the CMEC and 
CESC, provinces are 
publishing student 
outcome indicators of 
achievement (SAIP!) and 
involvement in education 
(participation, graduation, 
dropout rates) (Portrait?). 


Provinces are beginning 
to share the cost of 
developing curriculum 
and assessment materials. 


Existing national surveys 
of K-12 education are 
either dated (1990) or 
developed for specific 
clients. 


The PCEIP? is developing 
an indicator of satisfaction 
with education in Canada. 


1 School Achievement Indicators Program 
24 Statistical Portrait of Elementary and ticles Education in Canada 
3Pan-Canadian Education Indicators Program 
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on administrative data and the last three on survey data. Not only do the methods 
of calculation provide different estimates of dropout rates (from 18% by method 5 
versus 32% by method 2 for Canada), but they also rank the provinces differently, 
and confuse everyone as to what the rates mean. When the rates can vary by as 
much as three times for a province (14% by method 5 and 40% by method 1 for 
Alberta), there is obviously a problem in how analysts have conceptualized and 
calculated this irdicator. Some of the difference in rates is accounted for by the 
fact that various methods address different aspects of the dropout problem, but 
limitations in some of the methods have resulted in misleading statistics (Alberta 
Education, 1994a). 


A provincial student-level data base allows provinces to calculate more precise 
estimates by actually tracking students over time.’ Calculations can be refined if 
the effects of interprovincial migration are addressed in the estimate. In the absence 
of a rational student-level data base, calculation of national dropout indicators is 
limiced by the types of data available for interprovincial comparisons. These 
calculations are then a compromise which provide estimates of provincial rates 
based on the national definition used, but frustrate provinces that can calculate 
more precise estimates. Since many educational organizations and the media use 
interprovincial comparisons to make statements about Canadian education, it is in 
the best interests of the provinces to work diligently toward providing comparable 
data that can provide the most precise estimates possible. 


Student Comparability — This can be a serious problem in international studies where 
mass and elite education systems are sometimes compared. In Canada, all students 
must remain in school until the age of 16, and indicators are based largely on the 
total student population. Provinces that have assessment and examination programs 
typically assess all students in selected subjects at specific grades, and sample ' 
students for performance tasks. Many provinces also provide analyses of results 

by gender and other subgroups. Results are then tracked over time. Nationally, 
representative samples are drawn from all provinces participating in the School 
Achievement Indicators Program. Common definitions are used in calculating 
g-aduation, participation, and dropout (school leaver) rates based on cohort 

analysis of provincial populations at specific ages. As mentioned above, however, 
the national definitions are not always those used at the provincial level where 
student-level data bases permit more accurate estimates. 
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Census or Sample - Most provinces assess all students through their assessment 
programs whose major function is monitoring performance; several provinces also 
draw representative samples of students to complete performance tasks. High school 
exit examinations certify student achievement; all students enrolled in these courses 
write the exams. Four provinces (British Columbia, Alberta, Saskatchewan, and 
Manitoba) sample students to collect their perceptions about aspects of schooling; 
three provinces (Ontario, New Brunswick, and Newfoundland) survey all of their 
students as part of their assessment programs. At the national level, all students in 
the target population are included in the calculation of graduation, participation, and 
dropout rates. For the SAIP assessments, representative samples of 13- and 16-year- 
old students are drawn, ages at which Canadian students are still in school. 


Student Data Bases — Provinces are either developing a student-level data base 
or already have one. British Columbia, Alberta, Quebec, New Brunswick, 
Newfoundland, and the Northwest Territories currently have student-level data 
bases. In most cases, the data base began with high school students because of 
the need to certify graduates when they leave school. A student-level data base 
facilitates greater precision in estimating student involvement indicators and 
analysis of comparisons among different variables. The biggest issue with a 
student-level data base is confidentiality. Provinces need policies to protect 
student-level data and to determine eligibility for access to such information. 
This is becoming more important with freedom of information legislation 
allowing the public to have access to aggregated school-level data. Since 
there is no national student-level data base for education, involvement 
indicators must be calculated on the basis of age or grade cohorts. 


Data Collection - Schools administer achievement tests to their students. New 
Brunswick identified four issues related to data collection: (1) burden on schools 
and districts; (2) perceived need for/relevance of new data, (3) reliability and 
validity, and (4) impact on financial and human resources. 


We make sure we have a really good reason for collecting data. The 
need is then balanced against factors such as how much effort (time, 
money, staff) will be required at the department, district, and schools. 
We must ask ourselves if the data can be collected within the current 
budget (as new money is hard to come by), can we obtain the data 
efficiently using new technologies, are standards in place to ensure 
that the data are reliable and comparable? 


Survey respondent from New Brunswick 
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At the national level, each province except Saskatchewan collects its own data for 
SAIP and the CESC portrait. Guidelines are in place to ensure that uniform data 
collection procedures are followed. CESC (1994) published a Handbook of 
Education Terminology Elementary and Secondary Level to standardize 
definitions and analytic applications. 


Data Analysis - Provinces analyze student achievement data themselves; British 
Columbia also uses contractors to help with the analysis. Saskatchewan indicated 
that it completes analyses in consultation with stakeholders. Alberta has engaged 
external consultants to collect and analyze perceptions about education; third-party 
administration and analysis enhance the credibility of survey findings. At the 
national level, Statistics Canada analyzes the data for the CESC statistical portrait. 
Teams of teachers from all participating provinces score the open-response 
questions of the SAIP assessments in central locations under the supervision of 
the development teams; CMEC then arranges with a province to analyze the data, 
and engages consultants to prepare the reports. Provinces have input into the draft 
report before publication. Table 5 summarizes the discussion of technical and 
administrative issues. 


Interpretive and Communicative Issues 

Whereas educators have had access to professional information about the education 
system for some time, parents and the public have only recently become target 
audiences for such information. These lay audiences are particularly interested in 
what students know and can do. Therefore, student outcome indicators need to be 
reported in such a way that diverse needs can be served: parents want to know that 
their children are learning, the public wants to know that its investment in education 
is providing desired results, educators want information to help them improve 
results, and so forth. A single report will not serve all of the different needs. The 
issues related to interpreting and communicating information for various audiences 
include validity and reliability, context and interpretation, district/school support, 
and public reporting. 


Validity and Reliability - Provinces are careful to ensure that their assessment 
instruments are valid and reliable. Instruments are usually criterion referenced to 
provincial programs of study, are reviewed by curriculum specialists and teachers, 
follow standard test development and field-testing procedures, and item analysis to 
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.| 5. Definitions 


6. Student 
comparability 


7. Census or sample 


8. Student data bases 


Table 5 


Provinces use their own 
definitions. Those with 
student-level data bases 
can calculate more precise 
estimates from adminis- 
trative data. 


Provinces are responsible 
for all students in their 
jurisdiction. 


Some provinces analyze 
results for subgroups. 


Provinces use a census for 
achievement testing and 
calculation of indicators 
from administrative data. 
They sample students for 
formance assessments. 

‘or opinion surveys, they 
either survey all students 
or sample them. 


Provinces either have or 
are developing student- 
level data to improve 
calculation of indicators 
and analysis of relation- 
ships among indicators. 
They are eres 
policies on confidentiality 
of student information 
under access to informa- 
tion legislation. 
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Summary of Technical and Administrative Issues and Canadian Solutions 


National Solutions 


Provinces have a common 
set of definitions for the 
portrait; they are beginning 
to discuss the value of a 
peel student-level data 


Use of common: 

¢ instruments for the 
SAIP assessments 

¢ definitions and methods 
for graduation, 
participation, 
dropout rates. 


Representative samples of 
students are included in the 
SAIP assessments; all 13- 
and 16-year-old students 
are in school. 


All students are included in 
the calculation of indicators 
from administrative data. 


Current involvement rates 
are based on age cohorts. 
CESC has common 
definitions for indicators 
and analytic applications. 


phe could gevee an 
“upon stan ora 
Siienievel data base and 
data exchange through the 
CESC. 


ke 


Uses and Issues 


Table 5 (continued) 
Summary of Technical and Administrative Issues and Canadian Solutions 


Technical and Provincial Solutions National Solutions 
Administrative 


9. Data collection Provinces collect their own | Provinces collect their own 
data. Schools administer | data for SAIP and CESC. 


provincial assessments, y 

and departments collect The CESC terminology 

administrative data. handbook helps to ensure 
uniform data collection 

procedures. 


CMEC arranges with a 
province to analyze data 
for the SAIP. 


Statistics Canada analyzes 
data for the portrait. 


10. Data analysis Provinces analyze 
assessment and 
administrative data 
in-house. 


ensure reliability. Teachers are extensively involved in the development of tests and 
in marking written response items. Test administration is standardized so that all 
students have the same amount of time to complete the assessments and follow the 
same directions.. Additional strategies include a provincial exam board (British 
Columbia) and the use of panels to set provincial standards for achievement tests 
(Alberta). 


Provincial assessments serve primarily as a way to monitor performance and 
evaluate programs, with no stakes for students. In the nine jurisdictions having 
high school exit examinations, the stakes for students range from 30 to 50 percent 
of the final mark in these courses, with the remainder assigned by teachers for 
class work; blended final marks are valid because they are based on both class 
work and the departmental examination. 


The SAIP assessments are developed in both English and French by teams of 
educators from lead provinces in consultation with subject specialists from each 
of the other provinces. Mathematics, reading and writing team members came 
from Alberta, Ontario, and Quebec; science team members are from Alberta, New 
Brunswick, Ontario, and Saskatchewan. Criteria are developed to identify and 
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describe five levels covering the expected range of achievement for both age groups 
in each subject area. Questions are then developed to measure attainment of these 
criteria. Teachers from across Canada participate in developing the test items. The 
development teams and subject specialists review the questions to ensure that they 
are free from cultural and gender bias and stereotyping. Each province also verifies 
that the assessment materials provide a good match with its curriculum objectives. 
Questions are extensively field-tested to confirm that the results could be used to 
assign students to levels. 


Context and Interpretation — Provinces are sensitive to the effect of context on 
student outcomes. Most survey respondents indicated that they use contextual 
variables to help explain or interpret outcomes. Strategies used to help interpret 
results include interpretation panels and response groups (British Columbia), 
guidelines (Alberta), consultations with stakeholders (Saskatchewan), and 
workshops and comprehensive reports (Newfoundland). New Brunswick 
indicated it publishes annual dropout rates by school but does not provide any 
interpretation of the results; districts receive their results in advance of the public 
which allows them time to prepare their own explanations of local results. 

Any indicator can be distorted unless there is an understanding of the 

contextual factors. There has to be an emphasis on the use of many 


indicators, both qualitative and quantitative, if judgments about the 
effectiveness of the system are to be reliable and valid. 


Survey respondent from Newfoundland 


At the national level, little interpretation of student outcome indicators is provided. 
The CESC portrait contains an appendix describing the financing of education in 
each province. The SAIP reports provide limited interpretation; in addition to the 
achievement results for Canada and each province and information about the 
assessment, they describe the social context, organization of the school system, 
and teaching and testing in each province. Provinces assume responsibility for 
discussing results when the reports are released since they are accountable for 
decisions regarding the scope and sequence of programs and resource allocations. 


Distric/School Support - Survey respondents interpreted this question in two 
ways. Some responded in terms of provincial support to districts and schools, 
while others described the support districts demonstrate for provincial indicators. 
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- The former indicated that they provide school and jurisdiction reports and 
2 - consultation with local staff regarding policy positions and measures (Alberta), 
, and professional development (Ontario). Newfoundland reported that the 
iW ry 4 comprehensive school profiles have elevated the need for. full district/school 
° support; it plans to introduce a local indicator report card pilot project at the 
district level, followed by the school level. New Brunswick k indicated that it 
does everything possible to reduce the burden, provide resouices, and demonstrate 
the value of indicators i in order to get district and school support. Nova Scotia 
involves districts and schools in planning and concentrates on how the information 
can help them. Upon request, Manitoba provides school divisions with data and 
analyses which correspond to their own division, as well as those for the province 
.* as a whole. 


. ‘ British Columbia requires districts to report to their publics annually, and, as of the 
"1993794 school year, these reports must contain ten indicators of performance (both 
cae student and district). The province provides the necessary data‘and a set of graphic 
* and page layout templates. As well, it provides schools and districts with annual 
- profiles that contain district data and a brief interpretive framework. 
; ay i 
-.'+ District and school support is important for the SAIP assessments because selected 
. schools must administer the tests to their students. Proviticial coordinators work 
-with these schools to monitor administration and to ensure that these schools 
, _, Support the program. Because the CESC portrait relies on provincial administrative 
". data, it requires the cooperation of departmental staff. 
9 
= Public Reporting - Provinces release results for their assessment programs 
- through annual reports. These reports are written for a professional audience and 
” \provide a provincial picture of student results.. Some also have newsletters for 
‘distribution to the publi¢, for example, How are Students Doing? (Alberta). 
4 oo acces and press releases are also targeted at the general public. 


Relevance and timeliness are hao in public reporting. Provinces usually release 
_ provincial results through press releases soon n after the tests have been scored, and 
e results ‘have been sent to districts and schools. Provincial reports take longer to 

Prepare and sda not usually appear until the sii year, this is tot unexpected 
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_. -by December of the year of testing. CMEC publishes 30,000 copies (22,500 in 
a’? *- English and 7,500 in French), distributes them widely, and makes them available 
=e sor.) “alpen request. A technical report is available: from each provincial ministry and 


~ is ie given the types of analyses these reports contain. The SAIP reports have appeared 


ay e "from CMEC within a year of test administration. ‘The CESC portrait is written for 
ty ; ; : "professional audiences.” The 1992 edition Presented data primarily for the 1989/90 
rags - "school year; ‘the 1995 edition will report data for the 1992/93 school year. The more 
Reon 2's" jurisdictions involved in providing data, the longer it takes to ip sealy the data and 
POL eee . 5 paliheeoet rae 

a 2 O. " Timeliness is a problem shared by the international indicator reports. The 1995 

ak ees Education at a Glance 3 contains data for the 1991/92 school year. In today's 

a 3 world'of the internet and live television coverage of events, people expect instant 
se eh es * Fesults. However, time is required to clean data, check ranges, verify entries, 


confirm data, and analyze results. As the collection and analysis of data become 

“+ more routine, this issue may be resolved. Table 6 summarizes the discussion of 

ee a * imtrprtive and communicative issues. 
“, Comparative Issues : : 

"Indicators take their meaning from comparisons with ray of iedemace - time, 
_, Broups, or targets. Eleven jurisdictions indicated that they track provincial trends 
” over time. Comparing performance at the local, provincial, national, or. *’ 
international levels varies from province to province.: For example, Canada is 

’ participating as a country in the Third International Mathematics and Science 
. ‘Study; five provinces are also participating as $ provincial samples - ~ British. 
‘ Columbia, Alberta, Ontario, New Brunswick, and Newfoundland. Statistics 
Canada prepares interprovincial comparisons of the OECD education indicators 
for the peteies: Targets include benchmarks and standards. Berichmarks 
describe the existing level (baseline) whereas standards define an optimal or 
"desired level; targets specify ‘a feasible level of improvement within a 
predetermined period of time. A target, gets an improve 14mt increment over what 
*- exists (the benchmark) i in an attempt’ to reach what is desired (the standard). Five 
_, Provinces indicated that provincial performance serves as a provincial standard. 
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Table 6 
Summary of Interpretive and Communicative Issues and Canadian Solutions 


Provincial Solutions National Solutions 


11. Validity and Provinces base their Teams develop SAIP tests 

reliability assessments on their in English and French in 
programs of study consultation with subject 
and follow standard specialists from each 
procedures in developing | province. Tests are 
instruments. reviewed for cultural 

; and gender bias, and 

Standardized procedures stereotyping. 
are in place for uniform 
test administration. Indicators in the portrait 


are based on agreed-up 
definitions and methods. 


12. Context and 


Provinces use contextual National reports contain 


interpretation information to help limited interpretation. 
interpret provincial Provinces interpret results 
results. according to their own 
circumstances. 


13. District/school District and school , ata peel opne tl oe 
support 8 is i t for work with sample sc: 
untform siatnigration of | for the SAIP assessments 
assessment instruments to ensure support. 
and use of results inl Privacil ar 
improve student ing. vinci sentatives 
es S| ‘provide administrative data 

Provinces assist districts or the CESC portrait. 


and schools to interpret 
and use local results. 


There are three types of 
provincial reports: 
® press releases for results 

soon after tests are scored 
* annual within a 

year of administration 
* public indicator reports 
Press releases are timely 
but apd little analysis 
while results reports take 
longer to prepare but 
ipa in-depth analyses 

or professionals. 


14. Public reporting SAIP results are released 


in a timely manner. 
CESC portraits are 
published biennially. 
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School/District Comparisons — Provinces provide results on assessments and 
examinations to their districts and schools for local use. Five jurisdictions 

indicated they avoid or discourage comparisons between schools and the district 
(Saskatchewan, Manitoba, Ontario, Northwest Territories, Yukon). The British 
Columbia Ministry of Education provides the results for a composite of comparable 
districts. Alberta Education encourages districts and schools to compare their results 
to provincial standards where 85% of students are expected to achieve the acceptable 
standard or higher on provincial achievement tests at grades 3, 6, and 9. Nova Scotia 
shares and discusses results and assists people in their interpretation and use of 
information. 


Three provinces publish results by district and/or school. Newfoundland provides 
district comparisons in provincial documents. Quebec has been publishing school 
board results for some time; in 1994 it began publishing results by school as well 
(Ministére de l'Education du Québec, 1994), New Brunswick publishes annual 
dropout rates by school. 


Interprovincial Comparisons — All jurisdictions reported using some or all of 
the national and international reports (Statistics Canada reports, the SAIP 
assessments, the CESC portrait, and IAEP II). See Table 2, p.7. Provinces are 
cooperating on national tests and expanding the range of outcome indicators 
through the development of the Pan-Canadian Education Indicators Program. 
The PCEIP is also involving national nongovernment associations to enhance 
its validity. Some provinces indicated they will use the TIMSS data and other 
sources for comparative purposes. Table 7 summarizes the discussion of 
comparative issues. 


Conclusion 
Since provinces are responsible for education in Canada, they set the direction 
for themselves and their constituents. Of the three types of student outcome 
indicators, cognitive ones predominate. Provincial student assessment and 
examination programs are the most common way for provinces to demonstrate 
accountability. Most provinces make use of one or both of these programs. 
In order to provide external points of reference (other provinces and Canada as 
a whole), provinces are cooperating through the Council of Ministers of Education, 
Canada in the development and implementation of the School Achievement 
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Table 7 
Summary of Comparative Issues and Canadian Solutions 


Comparative Issues Provincial Solutions National Solutions 


15. School/district Provinces take three Not applicable. 
comparisons general approaches: 
* some enco' 
comparisons with 
provincial results 
* some discourage 
school/district 
comparisons 
* some publish school 
and district results 


16. Interprovincial Interprovincial Provinces are cooperating 
comparisons comparisons provide through CMEC and CESC 
external points of to provide comparable data 
reference. on selected indicators. 
Most provinces use inter- 
pone data provided 
y: 

¢ Statistics Canada 
¢ SAIP 


° CESC portrait 
* IAEP Il 


Indicators Program (for mathematics, reading, writing, and science), and through 

the Canadian Education Statistics Council in the Pan-Canadian Education Indicators 
Program (academic achievement, accessibility, student flows, school/work transitions, 
citizenship, and satisfaction). Periodic international assessments provide a way to 
compare Canadian education systems with those in other countries. 


Cognitive indicators are the most commonly used student outcome indicators in 
Canada as they are elsewhere. Most provinces also use graduation and dropout 
rates; seven collect information on affective indicators as well. Eleven jurisdictions 
use indicators to track trends over time, ten use them for public accountability, and 
nine for informing policy and practice, decision making, and program evaluation. 
Eight use them for program improvement and strategic planning. Provinces provide 
information to local jurisdictions, but vary in how they want districts and schools to 
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use provincial, district, and school results. All types of information are provided to 
help improve student learning; beyond that, provinces follow different approaches 
in their expectations of district and school use and dissemination of information. 


Political and financial, technical and administrative, interpretive and communicative, 
and comparative issues are common to most indicators. Provinces resolve these 
issues in similar ways and cooperate through the CMEC and CESC to address them 
at the national level. The discussion of some of the indicators highlights the 
dilemmas inherent in varying provincial capacities to collect and analyze data. 

The pitfalls of alternative approaches, with their conflicting messages, undermine 
public confidence in the information they are intended to convey. 


Provinces, CMEC, and CESC are becoming more proactive in providing timely and 
relevant information to a public bombarded by the media's opinions and often 
inaccurate facts about the state of education. If public education is to enjoy 
taxpayers’ support in times of shrinking resources, public confidence is essential. 
Student outcome indicators focus attention on student performance, communicate 
how successful the performance is, and, in conjunction with an integrated indicator 
system, propose strategies for improvement. However, the emphasis must remain on 
student outcome indicators because they are fundamental to our efforts to improve 
student learning. When such indicators are based on the important goals of 
schooling, they become a powerful way to concentrate action on improvement. 
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Notes 

1. This special issue icatures accountability programs in British Columbia, 
Alberta, Ontario, Quebec, and Newfoundland. The articles are followed 
by comments from representatives of four national organizations — the 
Canadian School Boards Association, the Canadian Association of School 
Administrators, the Canadian Teachers’ Federation, and the Canadian 
Society for the Study of Education. The five provincial representatives 
then respond to the observations of the four discussants. 


2. Barlow and Robertson address the following myths associated with how our 
schools have failed us: at least 25% of Canadians are illiterate; our dropout 
rate is at least 30%; we spend more on education than [virtually] any 
country in the world, and have less to show for it; students in all countries 
with which Canada competes work harder and longer; when our student 
results are compared with those of our international competitors, we are 
beaten by nearly everybody. 


3. There is a wide range of full-text online newspaper and newswire data bases 
available in Canada and for Canada. Some examples can be described as follows: 
Canadian Press (CP) is cooperatively operated by Canada's newspapers and 
provides exhaustive coverage of social, political, economic, and cultural news 
events from bureaus across Canada, the United States, and the United Kingdom; 
Stories and updates are prepared for over 90 subscribing newspapers. Broadcast 
News serves more than 500 Canadian radio and television stations. Reuter's 
Textline Global News contains articles from all the major daily newspapers of 
Europe and other sources from the Middle East, Africa, Australia, New Zealand, 
and the Americas. 
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The search strategy to identify news items on education was as follows: FIND 
education OR teacher(s) OR school(s)(ing) wherever they appear in the headline 
or lead of the news items. It is estimated that approximately 20% of all stories 
retrieved in this manner on a daily basis do not deal with education in a prominent 
manner where the three key words (education OR teacher(s) OR school(s)(ing)) 
can create false drops (e.g., Harvard Business SCHOOL or EDUCATION of 
consumers). Taking this into account, this search strategy identified 753 stories 
from Canadian Press Newswire Service, 781 from a global search of Canada's 
full-text online newspaper and newswire data bases, 575 from Reuter's Textline, 
and 1,223 from a local/regional clipping service in Alberta. We estimate the 
duplication among data bases to be about 10% between CP and the Alberta 
service, and 10% between CP and the global search. 


The Human Development Index (HDI) is a composite of three basic 
components of human development: longevity (life expectancy), knowledge 
(a combination of adult literacy and mean years of schooling (2/3 and 1/3 
weight respectively)), and standard of living (purchasing power, based on 
real GDP per capita adjusted for the local cost of living (purchasing power 
parity, or PPP). The HDI offers an alternative to GNP for measuring the 
relative socioeconomic progress of nations and enables people to evaluate 
progress over time and to determine priorities for policy intervention (United 
Nations Human Development Programme, 1994, p. 91). 


Finn (1994) criticizes the HDI because it is based on only three factors (the 
only three that could be measured for all 173 countries) and that it reflects 
national averages, thus failing to reveal inequalities that exist within the 
countries studied. Castonguay (1995) criticizes the index because it clusters 
the top 17 nations since the index estimates adult literacy in the top 17 
countries to be 0.99, and the income index fails to differentiate significantly 
among the leading countries; this leaves only life expectancy and schooling 
to weakly distinguish among the top 17. Both critics recommend the use of 
the HDI adjusted for gender disparity and income distribution. Canada ranks 
9th on the HDI adjusted for gender disparity (Sweden is first) and 8th on the 
HDI adjusted for income distribution (Japan is first). 
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